emcf / thepipe Goto Github PK
View Code? Open in Web Editor NEWExtract markdown and images from PDFs, URLs, docs, slides, and more, ready for multimodal LLMs. ⚡
Home Page: https://thepi.pe
License: MIT License
Extract markdown and images from PDFs, URLs, docs, slides, and more, ready for multimodal LLMs. ⚡
Home Page: https://thepi.pe
License: MIT License
I was looking at your pipeline and thought you might be better served by using https://github.com/Vaibhavs10/insanely-fast-whisper or allow a bit of wiggle room in your framework to allow an optional parameter for feeding in a seperate processor for video transcription problems. This is over an order of magnitude improvement on vanilla whisper and has cpu/gpu modes. You may want to just allow a whole pipeline to be fed to futureproof this particular endpoint to new tooling
Looking to support mp3
, wav
Audio is not standard in commercial multimodal models today in 2024. Because of this, I am also looking to transcribe audio to text, probably via Whisper.
error 'NoneType' object has no attribute 'write_audiofile'
occuring on line video.subclip(start_time, end_time).audio.write_audiofile(audio_path, codec='pcm_s16le')
Could probably fix with a simple none check
audio = video.subclip(start_time, end_time).audio
if audio is None:
transcription = None
else:
audio.write_audiofile(audio_path, codec='pcm_s16le')
...
If Tesseract OCR is not installed correctly, image extraction with text_only=True will yield tesseract is not installed or it's not in your PATH. See README file for more information.
. This occurs with improper Tesseract installation, such as the case with the current GitHub actions CI setup for this repo.
Further discussion here: https://stackoverflow.com/questions/50951955/pytesseract-tesseractnotfound-error-tesseract-is-not-installed-or-its-not-i
Multiple Questions:
What are the resources recommend/required for local extraction?
When running locally can you provide us the option to expose a port and receive POST requests? That way we can have an on prem machine that can work interchangeably with your API for client machines.
thepipe https://www.linkedin.com/in/spencer-reitsma-8a3938151/
Extracting from website...
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Python312\Scripts\thepipe.exe_main.py", line 7, in
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\thepipe.py", line 60, in main
chunks = extractor.extract_from_source(source=args.source, match=args.match, ignore=args.ignore, limit=args.limit, verbose=args.verbose, ai_extraction=args.ai_extraction, text_only=args.text_only, local=args.local)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\extractor.py", line 57, in extract_from_source
return extract_url(url=source, text_only=text_only, local=local)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\extractor.py", line 292, in extract_url
raise ValueError(f"{response['error']}")
ValueError: Page.evaluate: Execution context was destroyed, most likely because of a navigation
PS D:\Downloads\Project Templates for reference only>
I have added the env var THEPIPE_API_KEY to my .env, .bashrc and at the commandline. It is not getting accepted.
{response['error']}")
ValueError: Valid environment variable THEPIPE_API_KEY not found. You may need to restart if you have set your API key. Visit https://thepi.pe/docs to learn more.
Add .ino functionality for GitHub repos related to arduino
Hi! Not sure if this is a bug or a feature, but I'd love to use the ai_extraction
option to improve the handling of PDF documents. However, enabling this option overwrites the local=True
option.
MWE:
from thepipe.thepipe_api import thepipe
source = 'example.pdf'
messages = thepipe.extract(source, local=True, verbose=True, ai_extraction=True)
Throws the error:
Failed to extract from example.pdf: No valid API key given. Visit https://thepi.pe/docs to learn more.
It works without enabling ai_extraction,
but I don't like that it adds every page as an image to the messages because this massively increases the token count for longer PDFs.
As a workaround, I adapted the extract_pdf
function only to extract PDF pages as images if the page contains an image. It would be great to have this as an option. (I know this approach is not optimal as it misses tables and some images containing only SVG objects; maybe a better option is possible only based on the fitz
library, but I am no expert in this package).
def extract_pdf(file_path: str, ai_extraction: bool = False, text_only: bool = False, verbose: bool = False, limit: int = None) -> List[Chunk]:
chunks = []
if ai_extraction:
with open(file_path, "rb") as f:
response = requests.post(
url=API_URL,
files={'file': (file_path, f)},
data={'api_key': THEPIPE_API_KEY, 'ai_extraction': ai_extraction, 'text_only': text_only, 'limit': limit}
)
try:
response_json = response.json()
except json.JSONDecodeError:
raise ValueError(f"Our backend likely couldn't handle this request. This can happen with large content such as videos, streams, or very large files/websites. Re")
if 'error' in response_json:
raise ValueError(f"{response_json['error']}")
messages = response_json['messages']
chunks = create_chunks_from_messages(messages)
else:
import fitz
# extract text and images of each page from the PDF
with open(file_path, 'rb') as file:
doc = fitz.open(file_path)
for page in doc:
text = page.get_text()
image_list = page.get_images(full=True)
if text_only:
chunks.append(Chunk(path=file_path, text=text, image=None, source_type=SourceTypes.PDF))
elif image_list:
pix = page.get_pixmap()
img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
chunks.append(Chunk(path=file_path, text=text, image=img, source_type=SourceTypes.PDF))
else: chunks.append(Chunk(path=file_path, text=text, image=None, source_type=SourceTypes.PDF))
doc.close()
return chunks
Looking to support extraction of mp4
, mov
, webm
, avi
files as well as youtube
for a Vision-Language model (not a video model)
Video and audio is not standard in commercial multimodal models today. Because of this, I am looking to transcribe audio.
currently ignore only accepts one file type to be ignored
Accepting requests features in this thread, please feel free to suggest!
The roadmap so far includes:
When you have a video over 60 seconds it splits it into chunks like:
[00:55.800 --> 00:59.960] insert text here
but for a video which was originally longer than 60 seconds it would be great to see:
[01:55.800 --> 01:59.960] insert text here
Directory extraction fails if one file or any files fail, it should output what it has been able to extract.
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Python312\Scripts\thepipe.exe_main.py", line 7, in
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\thepipe.py", line 60, in main
chunks = extractor.extract_from_source(source=args.source, match=args.match, ignore=args.ignore, limit=args.limit, verbose=args.verbose, ai_extraction=args.ai_extraction, text_only=args.text_only, local=args.local)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\extractor.py", line 53, in extract_from_source
return extract_github(github_url=source, file_path='', match=match, ignore=ignore, text_only=text_only, verbose=verbose, ai_extraction=ai_extraction, branch='master')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\extractor.py", line 395, in extract_github
files_contents = extract_from_directory(dir_path=temp_dir, match=match, ignore=ignore, verbose=verbose, ai_extraction=ai_extraction, text_only=text_only)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\extractor.py", line 195, in extract_from_directory
for result in results:
File "C:\Python312\Lib\concurrent\futures_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python312\Lib\concurrent\futures_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "C:\Python312\Lib\concurrent\futures_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Python312\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self.exception
File "C:\Python312\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\extractor.py", line 194, in
results = executor.map(lambda file_path: extract_from_source(source=file_path, match=match, ignore=ignore, verbose=verbose, ai_extraction=ai_extraction, text_only=text_only, limit=limit, local=local), file_paths)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Spenc\AppData\Roaming\Python\Python312\site-packages\thepipe_api\extractor.py", line 46, in extract_from_source
raise ValueError(f"Could not detect source type for {source}.")
ValueError: Could not detect source type for C:\Users\Spenc\AppData\Local\Temp\tmptlxpz0r\Changelog.rst.
C:\Users\Spenc>
I'm running thepipe locally to extract some page URLs for processing with GPT4o, and it seems that the image generated for each page only captures the content above the fold (See example below). Is there a method to have it capture the entire page to be processed? (perhaps an argument such as fullPage=True/False)
My token limit for GPT4o as part of my plan is 10M, so I'm not overly concerned with hitting limits.
Example image: https://imgur.com/a/a06g3lh
Thoughts on a scan feature that prints file types of the directory/file selected for Piping without extracting any data? It would be clearer what file types are causing failure if something isn't supported.
Apologies there is no docker image yet! 😅
I am on the case.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.