The pipe is a multimodal-first tool for flattening entire directories and websites into a prompt-ready format for use with large language models. It is built on top of dozens of carefully-crafted heuristics to create sensible text and image prompts from files, directories, web pages, papers, github repos, etc.

• Prepare prompts from dozens of complex file types 📄
• Visual document extraction for complex PDFs, markdown, etc 🧠
• Outputs optimized for multimodal LLMs 🖼️ + 💬
• Auto compresses prompts over your set token limit 📦
• Works with missing file extensions, in-memory data streams 💾
• Works with directories, URL, git repos, and more 🌐
• Multi-threaded ⚡️

To use the pipe with Python, simply append the output to the start of your prompt:

import openai
import thepipe
openai_client = openai.OpenAI()
response = openai_client.chat.completions.create(
    model="gpt-4-vision-preview",
    messages = thepipe.extract("example.pdf"),
)

To use The Pipe, you will need playwright, ctags, pytesseract, and the python requirements:

git clone https://github.com/emcf/thepipe
pip install -r requirements.txt

Tip for windows users: you may need to install the python-libmagic binaries with pip install python-magic-bin.

Now you can use The Pipe:

python thepipe.py path/to/directory

This command will process all supported files within the specified directory, compressing any information over the token limit if necessary, and outputting the resulting prompt and images to a folder.

Arguments are:

• The input source (required): can be a file path, a URL, or a directory path.
• --match (optional): Regex pattern to match files in the directory.
• --ignore (optional): Regex pattern to ignore files in the directory.
• --limit (optional): The token limit for the output prompt, defaults to 100K. Prompts exceeding the limit will be compressed.
• --mathpix (optional): Extract images, tables, and math from PDFs using Mathpix.
• --text_only (optional): Do not extract images from documents or websites. Additionally, image files will be represented with OCR instead of as images.

You can use the pipe's output with other LLM providers via LiteLLM.

The pipe is accessible from the command line or from Python. The input source is either a file path, a URL, or a directory (or zip file) path. The pipe will extract information from the source and process it for downstream use with language models, vision transformers, or vision-language models. The output from the pipe is a sensible text-based (or multimodal) representation of the extracted information, carefully crafted to fit within context windows for any models from gemma-7b to GPT-4. It uses a variety of heuristics for optimal performance with vision-language models, including AI filetype detection with filetype detection, AI PDF extraction, efficient token compression, automatic image encoding, reranking for lost-in-the-middle effects, and more, all pre-built to work out-of-the-box.